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ABSTRACT 

An evaluation was done of the first year of the 
Undergraduate Research Opportunities Program (UROP) at the University 
of Michigan (Ann Arbor), which is designed not only to teach students 
about research and/or certain academic topics, but also to facilitate 
the identification of minority students with the university. This 
second aim is based on the current thinking that retention of 
minority students is significantly influenced by the social factors 
of identification with the university community. Failure to achieve 
identification with i^e institution may particularly plague minority 
students. The UROP pairs students with faculty advisors on the basis 
of mutual interests. The students then serve as research assistants 
to their faculty advisors throughout the course of the academic year, 
participating in laboratory groups, research discussions, and team 
meetings, '^he evaluation surveyed sophomores after their freshman 
year in the program and used an experimental condition (120 
under-represented minority students), a primary control condition (60 
African American UROP applicants), and a secondary control condition 
(146 white dorm residents). Results reveal that sophomores in the 
UROP were taking significantly more upper level credits than were 
either their minority student counterparts or the majority control 
students. Students also showed positive motivational and behavioral 
changes. Ten figures are included. (JB) 
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In today's talk I will be discussing the evaluation of a program that is very much the result 
of a team effort. The program was designed, implemented, and supervised by John 
Jonides in his capacity as Associate Dean of Research in the College of Literature, Science, 
and Arts at the University of Michigan. Jennifer Lemer was the program coordinator, and 
was responsible for the day-to-day operations of the program. She was also an important 
contributor to its evaluation. Biren Nagda was a reseai'ch associate on the program, and 
facilitated both the operation and the evaluation of the program. My role was simply that of 
an evaluator of the program, and together we designed and implemented the various 
conditions and procedures that I will be discussing today. 

As I'm sure you are all aware, minority education in the United States is beset by a host of 
problems. African- American students, for example, are more than twice as likely to drop 
out of college than Caucasian students. And this trend exists even among African- 
American students who perform outstandingly in high school. Because African- Americans 
are unlikely to begin college in the first place, the cumulative effect of these problems of 
recruitment and retention is that there are deplorably few African-Americans on our college 
campuses. To illustrate the scope of this problem, there are currently more African- 
American males in prison than in college. 

As you will note over the course of this talk, our orientation to this problem is a social one. 
We have begun with the perspective that although minority achievement may be beset with 
problems of remediation (perhaps originating from poorer schools, lower socio-economic 
status, etc.), social psychological factors are aiso a major barrier to minority academic 
achievement. In particular, our perspective has its roots in tlie research of Ogbu, Steele, 
Crocker, and their various colleagues, which proposes that minority students are frequently 
less identified with academic institutions than are their majority-student counteipaits. 
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These researchers have ai'gued that social factors that indicate identification with college are 
often more important predictors of academic outcomes (such as college retention, choice of 
major, and GPA) among minority students than are more traditional preparedness factors, 
such as SES, high school GPA, SAT, etc., which are commonly used to predict success 
among majority students, in response to their research on the importance of identification 
processes in the academic success of minority students, we have adapted to our own needs 
the type of undergraduate research program that is used at many universities to facilitate the 
matriculation of their students in gi'aduate programs. 

In our program, incoming minority students, and minority suidents who had just finished 
their first year, were solicited to join an undergraduate research oppoitunities program, or 
UROP. The goal of tliis program is not only to teach these students about research and/or a 
certain academic topic, but also to facilitate the identification of minority students with the 
university. This facilitation in identification is expected to emerge through the regular 
interaction between faculty and student paificipants in an atmosphere that is focussed on 
learning, but that is a highly social and personal one as well. In essence, we hope that the 
students' experience in UROP v/ill personalize, individualize, and humanize the academic 
institution in a way that classes and tutorials just cannot. 

Students in the program participate for either pay or credit, depending on their 
circumstances, and are pau*ed with a volunteer faculty advisor in the first weeks of the 
academic year. This pau*ing of student and advisor is based on mutual interests. The 
students tlien serve as research assistants to their faculty advisors throughout the course of 
the academic yeai'-participating in lab groups, research discussions, team meetings, etc. 
UROP maintains a small staff of peer counselors, typically previous graduates of the 
program, who meet regulariy with the participants and help with any difficulties that arise. 
On average, students in the program report tliat they spend 12 hours per week working on 
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their research activities, of which time 25% was spent woi'king with their academic 
advisor, 15% was spent with a graduate student assistant, 10% was spent with an 
undergraduate assistant, and 50% was spent working on their own. 

With that brief description of the nature of the students' participation, as well as our 
perspective and goals, let me turn now to the methods that we are using to create the 
appropriate control groups, and to collect the appropriate data in order that we might 
measure the efficacy of the program. 

One of the pnmary difficulties in assessing the effectiveness of intervention programs such 
as our own lies in the self-selection bias inherent to the participating sample. Students 
typically volunteer to participate in these programs for idiosyncratic reasons, and thus there 
are likely to be initial differences between the students who volunteer for the program and 
the students who do not-before any intervention has taken place. Even if a control sample 
is created that perfectly matches the intervention sample on all relevant demographic 
variables, such as SES, GPA, SAT, etc., there may be import^mt psychological differences 
between the samples (such as differences in motivation, interest, willingness to try 
something new, etc) that cannot be appropriately measured or equated. For this reason, 
when and if the participants out-perform the non-participants on any relevant dimensions, it 
is impossible to know whether the program facilitated their performance, or whether the 
participants were so strong to begin with that they would have out-peiformed the non- 
participants anyway. Under such ch*cumstances, it is even possible that the intervention 
program has had a debilitating effect, but that the participants were initially so much 
stronger than the non-participants that they out-performed them despite their participation in 
the program. 

This self-selection problem is not insurmountable, however, and with the appropriate 
assignment procedui'es and control groups, relatively strong inferences concerning the 
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efficacy of the intei^vention program can be made. Since the students who volunteer for a 
program can never be equated with those who do not, the most important procedure ^hat 
can be enacted is to randomly assign volunteers to participate in the program or to remain in 
a control group. Without such random assignment of volunteers, causal statements about 
the efficacy of the program can never be confidently made. Furthermore, it is vital that the 
students who are and who are not selected for the program be told that the selection process 
was essentially a lottery. Otherwise, any differences that emerge between samples could be 
a consequence of the early success experience of being chosen for the program on the part 
of some students, and the early failure experience of being rejected from the program on the 
part of other students. 

For the evaluation of UROP, we have an experimental condition, a primary control 
condition, and a secondary control condition. The experimental sample is composed of 
120 under- represented minority students, who have applied to the program to paiticipate 
for either academic credit or pay. These students were drawn from all of the eligible 
volunteers in the first and second year class, who were solicited during the summer before 
classes started. The 120 students who compose this condition were selected randomly 
from all of the eligible students who applied, and the 91 who were not selected sewed as 
the primary control sample, 

put up figure 1 here 

Because the students in the experimental condition were randomly selected from those who 
applied, they are as similar as possible to their primary control counterparts. The matching 
of these groups was facilitated through a paired-random assignment, in which students 
were paired on the basis of their high school CPA and the high school they attended. One 
member of each pair was then assigned to the experimental condition, and one member was 
assigned to the control condition. Such paired random assignment is unnecessary with 
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very large intei*vention programs, but it can substantially reduce the error variance that is 
typically present in smaller and more moderate sized intei^vention programs like our own. 

A secondary control group was also created, in order to establish an additional, baseline 
comparison sample of students who are representative of majority students in tlie incoming 
freshman class. The primary control group is necessary because it is a perfect match for 
the students in the program, but a secondary control sainple is also helpful in order to 
compare the psychological and behavioral changes in the experimental and primary control 
groups witli the changes that their Caucasian cohort ai*e simultaneously experiencing. The 
secondary control group was composed of 146 Caucasian students who live in one of the 
major undergraduate dormitories. These students were selected at the same time as the 
UROP sample, and their attitudes and academic outcomes were measured at the same time 
as well. 

To recap, the primaiy control condition enables us to detennine which effects are caused by 
participation in UROP and which effects are simply a function of the development and 
maturation of the students who applied to the program. Random assignment to this control 
condition enables us to partial out the extent to which the students who applied to UROP 
could be more academically motivated than students who did not apply to UROP. The 
secondary control condition allows us to compare th'^ development in identification, self- 
esteem, etc., of the UROP participants with Caucasian students who began college at the 
same time. The inclusion of this secondary control sample enables us to discern how the 
stud'^nts who apply to UROP differ initially and in their development from the Caucasian 
student population. Furthermore, we can examine the extent to which attitudes and 
identifications are differentially predictive of academic outcomes for the experimental, 
primary, and secondary control groups. 
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All of the students who were selected for paiticipation, whedier in the experimental, 
primary, or secondary control conditions, were administered surveys at the beginning and 
end of each academic year. The goal of these surveys is to measure the psychological 
changes that are brought about by participation in UROP, as well as by maturation and 
experience in college. We are then able to assess the extent to which these psychological 
changes are important in facilitating or inhibiting positive educational outcomes. 

While the control conditions allow meaningful interpretation of our data, we can also 
conduct mediational analyses that enable us to determine which psychological changes the 
program brings about, and which of these changes are instrumental in facilitating academic 
performance. Because we are proposing that students' experiences in UROP shouM lead 
to greater identification with college, and that this identification in turn should lead to 
improved academic performance, our design is focused around mediational measures 
collected over time. These measures will be collected annually for the next five years. 
Outcome measures such as grade point average, course selection patterns, etc., will also be 
measured at the end of each semester Our goal is to analyze these mediational and 
outcome data with regression-based causal modelling, as well as structural equations 
modelling. To this end, we are collecting multiple indicators of all of the primary 
mediational measures, both in the form of traditional scales, and in the form of newly 
developed dependent measui'es. 

To give you an example of the mediational model ihat we have in mind, here is a 
representative diagram of a path analysis of the effects of UROP, race, and social and 
academic preparedness on identification with college, the self-concept, and academic 
peiformance. 

Put up figure 2 here 
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Such an analysis will allow us to detennine the extent to which participation in URC'P 
facilitates academic peifonnance, and impoitaiitly, the extent to which this facilitation is 
mediated by changes in identification with college and the self-concept. Our prediction is 
that the mediated paths will carry a significant amount of variance, while the direct paths 
may or may not. 

These measures, models, and analyses will enable us to determine not only whether UROP 
facilitates academic performance, but if go, how UROP facilitates performance. We will be 
able to determine whether students in UROP get better grades and take more adviinced 
courses, whether they become more identified with college, and whether this increased 
identification causes the facilitation in academic performance. Only when multiple 
indicators are measured, over time, with groups that are randomly assigned, can such 
causal flow be established. Unfortunately, I will not be discussing the results cif these 
mediational analyses in today's talk, as these analyses require data collected over a longer 
period of time than just the first year of the program in order to determine wheiher 
attitudinal changes are leading to positive academic outcomes. 

The academic outcome measures that I will be discussing today, and that are of primary 
importance, are grade point averages and course selection patterns. I will not be discussing 
retention rates, important though they are, because at this early point in the program, very 
few students have dropped out of college. It is quite possible that when we examine the 
return rates this fall, we will begin to see differences in the retention rates of program 
participants and their primary control group. 

The attitudinal and behavioral scales that I win be discussing today are; self-esteem, 
collective self-esteem, styles of coping with academic challenges, number of university 
events attended, students' estimated probability of obtaining an advanced degree, and their 
confidence in their ability to conduct a research project. The justification for these 
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questions and scales is relatively straight-forwai'd, so let me briefly review our ideas behinu 
the administration of each of these scales. 

put up figure 3 here 

We are giving the Rosenberg self-esteem scale because recent research in social psychology 
has demonstrated that self-esteem scores are typically just as high among under-represented 
minority students as they are among majority students. Among minority students, 
however, self-esteem is not strong?y correlated with academic achievement, while it is 
highly correlated with academic achievement among majority students. The explanation for 
this difference is that minority students are thought to dis-identify with the academic 
system, which results in their self-esteem being protected while then* performance suffers. 

We hope to show that through paiticipation in UROP, minority students begin to develop 
an increased correlation between their self-esteem and their acadeinic perfoirnance. Such 
an increase in this correlation has the potential to lead to a momentary decrease in self- 
esteem, as students begin to feel that theu' academic performance is self-defining. Such a 
threat to self esteem that is posed by potentially poor academic performance should be 
offset, however, by the increase in skills and confidence that students gain through their 
participation in UROP. Consequently, self-esteem may actually rise among participants in 
the program, even as it becomes more tightly linked to academic performance. 

We are giving Crocker's collective self-esteem scale because it has been shown to serve as 
a buffer against the negative effects of perceptions of prejudice and barriers. Furthermore, 
it is our hope that UROP will either increase collective self-esteem or at least leave it 
unchanged. Our concern with collective self-esteem biems in part from recent research in 
psychology, sociology, and anthropology that has demonstrated that high levels of African- 
American academic achievement are often associated with a lack of ethnic identification. It 
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is our hope that UROP will facilitate the academic performance of UROP participants while 
still maintaining high levels of collective self esteem, or esteem in their ethnic identity. 

We are also giving some new measures that we have developed with die goal of measuring 
changes in identificauon with college, and behaviors that are relevant to this constnact. So, 
we are giving a scale that measure styles of coping with academic challenges. Previous 
research has indicated tliat minority students ai'e often less likely to seek help when tliey are 
not perfonning well. Our hope is that paiticipation in the program will make students more 
likely to seek out the help of their teaching assistants and professors when they do not 
understand the material, or when they do not do well on an exam or paper (for lack of a 
better name, I am referring to this concept as academic coping styles). We are also giving 
a scale that assesses the number of university events that the students attend each month. 
We reasoned that if minority students are less identified with college than their majority 
student counterparts, they would probably be less likely to attend university functions that 
are not required of them. We hoped that by virtue of their participation in the program, 
minority students would become more identified with college and begin attending more 
events. This increased attendance, in turn, should lead to even greater levels of 
identification, as minority students become involved in a variety of collegiate activities- 
both academic and social. Consequently, this scale measures attendance at both academic 
and social functions. Finally, because the program is research oriented, we wanted to 
measure the students' estimated probability of obtaining an advanced degree, and their 
confidence in their ability to conduct a research pro/ect. Both of these estimates should 
increase as a function of their participation in UROP. 

As I stated earlier, all of the attitudinal changes that I've discussed here should be 
instrumental in facilitating academic outcomes. I will not be presenting any analyses that 
assess the mediating roles of these attitudes on academic perfoimance, however, as we 
have only measured the effect of the program on these attitudes and on academic outcomes 
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during a single, end-of-the-year sui-vey. The attitudinal changes that I am reporting today 
should lead to facilitation in academic peifomiance in the years to come. 

Let me turn now to the results from our evaluation of the first year of UROP. 

put up figure 4 here 

To begin witli the attitudinal data, I have here a graph of tlie different attitudinal and 
behavioral data that I just described. In order to present tliese scales in a single graph, I 
have computed the mean response for each question in the scale, rather than summing the 
responses to the various questions as is usually done. So, for example, on the self esteem 
measure, subjects in the minority control condition had an average score of 3,33. Since 
this is really a ten item questionnaire, this score translates to a 33.3. In the same way, 
subjects in the UROP condition had an average of 35.2, and subjects in the majority conO^ol 
condition had an average score of 32.2. For all of the scales that you see on this graph, the 
overall main effect for condition was significant at the .01 level. 

As you can easily see in this graph, students in UROP show higher scores on every one of 
the dimensions than students in either control condition. Although the simple comparisons 
between conditions were not significant for each variable, the pattern of means across 
variables is essentially identical. In particular, students in UROP show higher levels of 
personal and collective self-esteem, they report more confidence in their research abilities 
and a greater likelihood of seeking an advanced degree, and they state that they are more 
likely to seek help from faculty and teaching assistants than students in either control 
group. 

If we turn to the number of university events that they attend each month, we again see the 
positive influence of the program, although the pattern is not quite the saine. 
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put up figure 5 here 

Not surprisingly, here you can see that majority students are attending a far greater number 
of university functions than are minority students. Nevertheless, the pattern of means once 
again reveals that UROP students are attending more university functions than their 
minority control counterparts, although in this case the difference is not significant 

Recall that I stated earlier that students in UROP might have lower self esteem than their 
minority counterparts, as the increased relationship between academic performance and self 
esteem might momentarily depress their self esteem scores. On the other hand, it was our 
hope that students in UROP would have equivalent or higher self esteem than their minority 
counterparts, as their increasing academic confidence, skills, and coping abilities might 
offset the potentially debilitating impact of their academic performance. Because tlie earlier 
results that I showed you (put figure 4 up again) suggest that self-esteem is slightly 
elevated by participation in UROP, we thought that this latter explanation might in fact 
prove to be tme. So, in order to examine the possibility that UROP provides the students 
with an attitudinal and behavioral repertoire that protects and strengthens their self esteem, 
we ran a path analysis in which we examined the impact of the program on the attitudinal 
and behavioral measures that I just showed you, and the impact of these measures, in turn, 
on self esteem. 

put up figure 6 here 

As you can see in Figure 6, the picture that emerged is exactly as we hoped. Participation 
in UROP has no significant dii*ect impact on self esteem, but it does have an indirect 
impact, primarily through its influence on the coping strategies and confidence it provides. 

So, the attitudinal and behavioral picture after a year in the program looks promising. 
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Our next goal was to examine the students' academic outcomes, to see if any differences 
between conditions have yet emerged. Tliere are two possible venues by which UROP 
might facilitate education: UROP might cause students to receive better grades, and UROP 
might induce students to take more classes and credits, especially at the more advanced 
level. An increase in CPA would obviously be desirable, and so long as an increase in the 
number of credits taken is not associated with a decrease in GPA, that would be quite 
desirable as well. 

To turn first to the students' grades, you can see that as of yet there is no unequivocal effect 
of the program on this variable. 

put up figures 7 and 8 here 

Students in UROP show functionally equivalent grades to their minority student 
counterparts, whether those grades ai*e received in lower level or upper level courses. Not 
surprisingly, the majority control students show the highest grades of these three groups in 
both lower level and upper level classes. This diff ^ence simply mirrors the national trend. 

If we then examine the number of credits that students are taking, the picture is more 
complex. 

put up figure 9 here 

Here we see that for lower level courses, once again die majority students are taking more 
credits than either group of minority students. These credits are received in 100 level 
courses, though, and thus are not of primary interest. 

When we turn to the number of upper level credits taken, we start to see a benefit of 
participation in the program. 
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put up figure 10 here 

Analyses of these results reveals that students in UROP are taking significantly more upper 
level credits than either their minority student counteiparts or the majority control students. 
This significant difference emerged no matter which variables were analyzed-either tlie 
number of classes or as the number of credits taken-and no matter how they were 
analyzed-either with no covariates, or with high school CPA, SAT, ACT, and previous 
courses taken as covariates. Importantly, and as you can see in the figure, the locus of this 
effect was entirely among sophomores. First year students who participated in the program 
showed no benefit whatsoever in the number of upper level credits taken. However, this 
interaction is quite probably the result of a floor effect on the part of the first year students 
(since so few of them take upper levei courses), and we are hopeful that the first year 
students will show the same gains next year that the sophomores shov/ed this year. For 
this reason, it is not possible to make program recommendations at this point. 

More micro analyses of these findings suggested that the increase in the number of credits 
taken resides primarily in the sciences, and there it resides predominantly in the social 
sciences (although to a significant degree in the natural sciences as well). Given the fact 
that most of the research placements in UROP are in the social sciences, it is both gratifying 
and unsurprising that this would be the primary locus of the effect. 

So what can we conclude about all this? First of all, the results I reported today are from 
just the first year of the program, so it is still too early to make firm conclusions about the 
efficacy of a program such as UROP. Nevertheless, the early returns are very 
encouraging. In particular, our data suggest that the progi'am is beneficial in three very 
important ways. 
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First, students in the program are showing positive motivational changes. They report that 
they are more likely to go graduate school than control students, and they have more 
confidence in their research skills. Furthermore, they show higher levels of self esteem 
than their minority counterparts. Thus, these changes in goals that the program brings 
about are not accompanied by the decreases in self esteem that are theorized to accompany a 
more academic locus of self esteem among minority students. In particular, the results of 
the path analysis suggest that participation in UROP protects self esteem by providing 
students with skills and confidence that offset the potential threat posed by their academic 
performance. Furthermore, the changes in goals brought about by the program are not 
accompanied by decreases in collective self esteem either. Previous research by Ogbu and 
others has suggested that African- American students who perform well academically, and 
are highly interested in academics, are often less ethnically identified than students who ai'e 
uninterested or perform pooriy. Our data suggest that African- American students can 
become more involved in academics without an associated cost in their level of ethnic or 
racial identification. 

Second, students in the program are showing positive behavioral changes. They report that 
they are more likely to seek help when they perform poorly in clas:^ than control students. 
Since previous research has suggested that minority students are often less likely to seek 
the help of faculty and teaching assistants than majority students, our data show that 
participation in a research program has the potential to reverse this debilitating reticence on 
the part of minority students. Although we cannot be sure why UROP students report an 
increased likelihood of seeking help when they need it, it seems plausible that participation 
in UROP personalizes the faculty, demonstrating to minority students that faculty are 
approachable, interested in, and supportive of the education of theii* minority students. 



ERLC 



i o 



UROP 
16 



Finally, and perhaps most importantly, students in tlie program are increasing the number 
of upper level courses that they choose to take. 7'hus, after only one year, the program is 
beginning to bring about positive educational outcomes. Once again, the locus of this 
effect is currently unspecified, but the program was designed to provide students with 
research experience that would spark then- interest in the kinds of topics that are the focus 
of advanced courses and seminars. To the extent that these minority students increase the 
number of upper level courses that they take, they are enhancing tlieh- opportunities during 
college and after they graduate. And because this increase in credit load is not associated 
with a connmensurate decrease in grade point average, it suggests that students in UROP 
are becoming more interested 'n academics and more dedicated to academics. 
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UROP Participants 

Total number = 120 

(complete data exist for 81 
African-American students) 



Primary Experimental 
Sample 



UROP Applicants 

Total number = 91 

(complete data exist for 60 
African-American students) 



Primary Control 
Sample 



Dorm Residents 

Total number = 199 

(complete data exist for 146 
Caucasian students) 



Secondary Control 
Sample 
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Figure 2 example of mediational path analysis 
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Figure 3 measures 

Self Esteem 

Collective Self Esteem 

Academic Coping Styles 

Probability of Obtaining 
an Advanced Degree 

Confidence in Research 
Abilities 

Attendance at University 
Events 



UROP 
20 



Figure 4 
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Figure 5 
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Figure 6 - path analysis on self esteem 




* p<.05 
** p < .01 
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Figure 7 
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Figure 8 



GPA in upper level courses 
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Figure 9 



Number of 100 leveJ credits earned - 1992 
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Figure 10 
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